Multi-modal Analysis of Interactional Rapport in Three Language/Cultural Groups

We describe studies underway of nonverbal and verbal aspects of adult human dyadic interactions in three language/cultural groups: Iraqi Arabic-, Mexican Spanish-, and American English-speaking cultures. Previous studies have identified cross-cultural differences in the diverse multi-modal behaviors, such as headnod, gaze and posture, that support interactional rapport by signaling engagement or endorsement. We present our corpus of more than 100 2to 10-minute, videotaped storytelling interactions. The speech data have been fully transcribed, aligned, and, for the Arabic and Spanish, translated. Nonverbal aspects of interaction are being annotated. The multi-modal analysis we present focuses on participants in the listener role. This is to inform later computer modeling of Embodied Conversational Agents (ECA), specifically, listening agents. Portions of this work have previously been presented at Interspeech 2010 and the 4th Conference of the International Society for Gesture Studies, 2010.